Skip to content

M7: Bumblebee conformance breadth — ViT + Whisper - #14

Merged
ausimian merged 6 commits into
mainfrom
m7-conformance-breadth
Apr 14, 2026
Merged

M7: Bumblebee conformance breadth — ViT + Whisper#14
ausimian merged 6 commits into
mainfrom
m7-conformance-breadth

Conversation

@ausimian

@ausimian ausimian commented Apr 14, 2026

Copy link
Copy Markdown
Owner

Summary

  • Extends M3 (DistilBERT) + M4 (Qwen3) conformance with ViT (vision, encoder-only, conv patch embed) and Whisper (audio, encoder-decoder, 1-D conv frontend, cross-attention KV cache). Four new tests total: tiny-random + full-checkpoint tier per model, matching the existing Qwen3 pattern.
  • Extracts Emily.ConformanceHelper (test/support/conformance_helper.ex) — setup_all backend swap + assert_all_close/3 — and migrates DistilBERT, Qwen3, ViT, Whisper onto it. Net change ~zero LOC today, terser for future suites.
  • Bug fix in Emily.Backend.via_binary: pin the default backend to Nx.BinaryBackend for the duration of the fallback fun. Surfaced by ViT tiny-random exercising conv: Nx.conv builds a zero-pad scalar internally, which landed on Emily.Backend whenever the conformance setup_all was active, then crashed inside BinaryBackend on the mixed-backend operand list. Never surfaced before because backend_fallbacks_test.exs doesn't install Emily as the global default.
  • PLAN.md renumbered: M7 = conformance breadth (this), M8 = native conv (lift Backend.conv off BinaryBackend), M9 = 1.0 release (was M7). MoE / Mixtral tracked as deferred pending upstream Bumblebee support.
  • test_helper.exs exclude list extended with :vit_full, :whisper_full (both opt-in via mix test --only <tag>).

Test plan

  • mix test — 183 tests, 56 properties, 1 doctest, 0 failures
  • mix test --only conformance — 16 tests, 0 failures (DistilBERT + Qwen3 + ViT + Whisper tiny)
  • mix test --only vit_full — 1 test, 0 failures (~21 s; pins argmax == 763 + leading-5 logits slice)
  • mix test --only whisper_full — 1 test, 0 failures (~7 min; pinned values verified on re-run)
  • mix format --check-formatted clean
  • mix credo --strict clean
  • mix test --only qwen3_fullnot re-run this session; the only touchpoint is the via_binary fix, which tightens a correctness case Qwen3 doesn't hit (Qwen3's hot path went native in M4). Worth a manual pass before merging if you want to be belt-and-braces.

Notes

Whisper full runs ~7 minutes on Apple Silicon because Emily.Backend.conv still routes through Nx.BinaryBackend — ~940M multiply-adds through BEAM loops across two encoder 1-D convs. M8 (wiring Backend.conv to the existing Native.conv_general) brings this to sub-second; intentionally out of scope for this MR to keep it focused on conformance.

DistilBERT and Qwen3 conformance suites each carry an identical
setup_all block (swap to Emily.Backend, restore on exit) and an
identical assert_all_close/3. Before adding ViT + Whisper suites
(which would be a third and fourth copy), lift both into
test/support/conformance_helper.ex. Net change ~zero LOC; keeps
future suites terse.
Ports Bumblebee.Vision.VitTest verbatim — three tiny-random
architectures (:base, :for_image_classification,
:for_masked_image_modeling), same synthetic pixel input, same
expected slices. First conformance suite to exercise the conv
fallback path in anger.

Surfaced a latent bug in Emily.Backend.via_binary/via_binary_tuple:
the helpers transferred input tensors to BinaryBackend but left the
global default backend as-is. When a fallback op (Nx.conv in this
case) constructs a scalar tensor internally — e.g. the zero-pad
scalar via `Nx.pad(t, 0, ...)` — that scalar landed on
Emily.Backend whenever the conformance setup_all had swapped the
global default. The resulting mixed-backend operand list crashed
inside BinaryBackend's op with a FunctionClauseError on
to_binary/1 (the Emily-backed scalar's `data` doesn't match the
`%{state: data}` pattern BinaryBackend uses for its own tensors).

Fix: pin the default backend to Nx.BinaryBackend for the duration
of the fun call in both via_binary and via_binary_tuple. Never
surfaced before because test/emily/backend_fallbacks_test.exs
doesn't install Emily as the global default — it constructs
operands with an explicit `backend: Emily.Backend` — and every
earlier Bumblebee suite (DistilBERT, Qwen3) had already moved its
hot-path ops off the fallback by M4.
Full-size ViT-Base forward pass on Emily.Backend, pinned against
a deterministic synthetic pixel input. Structure mirrors
qwen3_full_test.exs: @moduletag :vit_full, excluded even from
--only conformance because the checkpoint is ~330 MB. Run with
`mix test --only vit_full`.

Uses a constant-gray pixel tensor rather than a checked-in JPEG
fixture so the repo stays free of binary test assets and the
featurizer preprocessing path doesn't enter the assertion surface —
the intent is to catch numerical drift on real-size weight tensors,
not verify image preprocessing. Pinned values: leading 5 logits +
argmax index 763 (ImageNet class "revolver") on the gray input.

Also update test/test_helper.exs to exclude :vit_full and
:whisper_full (landing next) from the default test run; expand the
comment to document each heavyweight tag and its cache footprint.
Ports Bumblebee.Audio.WhisperTest verbatim — two tiny-random
architectures (:base, :for_conditional_generation), same synthetic
mel features (Nx.sin(Nx.iota({1, 60, 80}))), same decoder inputs,
same expected slices.

First conformance suite to exercise encoder-decoder cross-attention
on Emily.Backend, and the first with strided 1-D conv in the
encoder frontend. Both pass first try against Bumblebee's
HF-Transformers-produced reference slices, which means the
cross-attention KV-cache update and the two stacked 1-D convs are
numerically consistent with PyTorch — despite conv still routing
through the BinaryBackend fallback (M8).
Full-size Whisper-tiny forward pass on Emily.Backend, pinned
against a deterministic synthetic 30 s mel window (sin(iota({1,
3000, 80}) * 0.01)) and a short special-token decoder prompt.
Asserts the leading 3×3 logits slice at 1e-3 tolerance plus the
decoder-last-step argmax (== 50257, <|endoftext|>, which is what
the synthetic input collapses to — that's fine for a pin; the
assertion is "same backend + same weights + same input reproduces
the same token", not "the model says something interesting").

Runtime is ~7 minutes on Apple Silicon because Emily.Backend.conv
still routes through Nx.BinaryBackend (deferred to M8). Whisper
encoder has two 1-D convs (kernel=3, out=384) over 3000 time-steps,
~940M multiply-adds executed as BEAM loops. The full test is
:whisper_full tagged, excluded from --only conformance, and opt-in
only; M8 wiring Backend.conv to Native.conv_general will bring this
to sub-second.

Also picks up formatter fixups on lib/emily/backend.ex (the
via_binary assignment from the previous commit) and trailing blank
lines on distilbert_test.exs / qwen3_test.exs.
- Renumber milestones: M7 = conformance breadth (this), M8 =
  native conv (lift Backend.conv off the BinaryBackend fallback),
  M9 = 1.0 release (was M7).
- M7 section documents the two new models and the rationale
  (vision, encoder-decoder, conv coverage) and records that
  MoE / Mixtral is deferred pending upstream Bumblebee support.
- RELEASE.md accumulates the added test suites, the shared
  ConformanceHelper, the test_helper exclude-list update, and
  the via_binary default-backend fix surfaced by ViT tiny-random.
@ausimian
ausimian merged commit 7ea032c into main Apr 14, 2026
1 check passed
@ausimian
ausimian deleted the m7-conformance-breadth branch April 14, 2026 12:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant